• 2 UK Patent Application „„GB ,,,,2327837 ,„A 





D«e ,i hUr.$ 03.07 
Priority Dirts 




m> 

{723 


MfETOSOft CofpOrStiSSK 

iirs*;s!$3orjst«fj ;« USA - Wa&rrmgtonS 

Otas teisrasvft SSedKKMWi, VVA 9S0S2-6333, 

United Ststss of Asnarfca 


;5fi; 0<:?vr.4nii. Gts'J 

m zarnru a 

iSSs f isid of Search 

UK CL iSciitiiw f ; H4F ?SS . H4T TSAX TB8& T»AA 
:NT CI" 

ONLINE. VOT, 3NT*«8£T 










Ager" am/of Address tor Ssir/tr 

4 DyWs, Syiidings. ►fcfcoro, lOaiOOft EC1N JjT, 
UnNttd Kingdom 




<:s4i 







If snsmtejsier!, reception and dispUy 05 corobmed video data in a s$t »f*a af 3 hyptfrfsnis dsta fii*. 

;5?j A video brcarftasr syswm ^c^d^s. 3 broadcast swurc? Wat bfosciss&is « vids?o s!f«sm and provides; 
atmivipanvlni} s:»ppi«fnerst3i d»« Stes. ^ach suppferfnΠ;! osts fiie is an HTML ftfe haying instructions for 
r«fid«:f:g 3 hv»fcH!r>k ov-jrisy oh the video ttr g3fyi A receiver =s coniigurs^ $0 receive sh>,? vkfiic; stsssro and 
accompany:^ suppietnsntai dsLs ?s : es. and -o disomy the hy&srifik ovsriays So conjunction vvtih She vidtsc 
stf t,-;jm The ov^-ays dashes fcavjrsQ D^k^oun.-H of 3 ore-cietsrn^nsd vaf-x- which is used ss a cofot key 
j;i t serving S3qu(ewt:?ii. Spswiteaiiy. me re .tewing equipment is sonfigyrsa to fersder video orsfy =n display 
areas ?hst srs s,at ro coior sey vshie. T;il;;;. the video sttesrp ;s< rendered 'hehirid* ths hypsfhfji: ovf-'isys. 
3f-« ;he hairs-grounds of She overlays aps>s Jfs transparent. C.fWfci <*sts «s provided with tne KTivfL flies, to 
sndicst* ^vhof; th& oysfisys; should he r^f.ctsrsci 3*-.o to pic,Vid« sji-jp? inspection?, on how ;h« HTML ?iie? shou d 
ise hanciisd by ;he receiyrcg sqiiiomeftt. 




sDEC 



PROGRAM 
MEMO 1 ?* 



TAPS ' 
DRSV£ 



VOUTHE MSMGRY ; 



AUOJG ; 

BOARD ! 



; U/ LPT 1 



. KEYBOARD j 







... — 220 




Transmit Video 
Stream 






Associate HTML. 
Fife With Video 
Stream 


T 








\ 

Formal HTML Flies 




Transmit HTML 
Flies and Control 
Files 













224 



4 



* -n 230 

Receive V a«?s> 

Sfear hTL M 
Files anc Cem-cl 



232 



Stcsrr Using CUcf 



Re no or HTML 
?v<*; ax « 



"pif* 5 



2327837 



PROVIDING SNHAHCSD CONTENT «ITH BROADCAST VIDEO 




This invention relates methods and systems for providing and displaying 



o Moreover, higher quality digital vjcco feeds distributed, for example, via Direct 

sc Broadcast Satellite (DBS;. <viU soon also be available on PCs. "With the right 

r. ! hardware, PCs will be able to display and decode DBS signals, analog cable 

t:i signals, and over-the-air-signals. as well as analog and digital video signals stored 

: i locally i on a CD or VCR, for example) . 

j Is the ruture, television signal decoder circuitry for both analog and digital 

»?i signals will become extremely prevalent, it' not standard, on all PCs- By enabling 

ii | a user or viewer to wors with, view, and use both data and video simultaneously, 

? ij ^e s d oet*e > e .set .. e* es\e- - ^perteree te\c< a a ^ ~~aro *e<e^ 

si la this new broadcast environment, television broadcasters will want to take 

:y advantage of PC capabilities by providing digital program enhancements. For 

| example, a broadcaster might want to provide ancillary statistics during a sports 

- broadcast, allowing a viewer to find more detailed information on a particular ream 

« or player. Similarly, & broadcaster might provide relevant infotroanon on the topic 
of a documentary, io that the viewer can access more detailed information either 

:« ;. during or after the broadcast. As another example, h might be desired to advertise 



; j ancillary data content with transmitted video and audio. 




$>j Displaying live video on a personal computer iPCi is becoming more and 

■.■j; more common. A number of low cost video capture boards allow tor display of 
•in analog video signals on the RGB display surface of a personal computer. 
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| program-related merchandise during a program — such as dolls or other to>s during 

:!; a children s cartoon program. 

:| This method of delivering data ro PCs is particularly attractive in areas of 

:| the world where there is not a great deal of traditional data infrastructure. For 

s ] example, consider India which has a middle class population of about 300 million 

is\ people (greater than the entire population of the US): while individuals, can afford 

■ | a PC and many do have PCs, few have experienced the Internet or high speed data 

%\ networking, because of underdeveloped telecommunications mnsstracrure. The 

s possibility of delivering data sendees to such locations presents a number of 

to opportunities. 

v. \\ Recent years have also seen many efforts to develop what, is referred to as; 

:2 ^interactive television." Interactive television allows a viewer to interact with a 

u cable television service provider. Various enhanced features are envisioned for 

:j interactive television systems. For instance, interactive program guides can foe 

;■:!; ~rn <-rientevt rot m^-vis ^ og an 'nteraeu^e projr^m gm^e i \'iev<.*' ca~ ^rrrf. 

| through television listings using cursor control buttons on a remote control. A 

:•. viewer can also perform interactive searches of program content. Some interactive 

, s TV systems have included the ability- for broadcasters to provide ancillary data 

ssi s| along with video feeds, 

:o | There has been no widespread acceptance of any interactive television 

- j system. One impediment to such widespread acceptance is that each system uses 

- ! different standards and protocols for providing, data that is ancillary to the primary 

i v;deo content. Untess a single system is insti rated as ;i standard, broadcasters will 

:.i \ have to provide ancillary data in numerous different formats. This is a significant 

:« ! impediment. 
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The mverttcn removes this impediment by uuinung conventional formats 
for providing ancillary data along with video broadcasts, along vcifo a scheme for 
overlaying digital data content on the prmaarv video stream. 

The invent-on allows video broadcasters to prepare ancillary data content as 
HTML files. The HTML files are prepared as overlays, with backgrounds of a 
pre-deterrruned color key tor viewing with video equipment having coior kevtna 
features. At a receiver, the HTML overlays are rendered using typical internet 
browser technology ;n the same display area as broadcast video, u&mg color 
keying. This makes the overlay background appear transparent: the video aooears 
only ;n the background areas of the ! 1 LM; ^vci.a\ 

Fig. l is a block diagram of a uJsj -roadcasr^ig i\s;err m accordance with 
the invention. 

Fig. 2 is a block diagram of *,\erap.«;v zece.vtr n accordance with the 
invenrioTt 

Fig. 3 shows how video. HI ML overlap a ad iiame windows are 
configured in accordance with the mvrmoa 

Figs i and 5 are flowcharts ^nowttg prster- ed methodological steps in 
accordance with the invention. 
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••\ Fig. ) sho^s 3 video broadcasting system in accordance with one 

4 embodiment of the invention, generally designated by reference numeral 10. 

Broadcasting system 10 includes a broadcast source 12 and a plurality of receivers 
?! 14, The broadcast source in this embodiment of the invention is a digital satellite 
-il system that broadcasts video content or, multiple RF frequencies or channels, 
r: Alternatively, the broadcast source might be a traditional terrestrial television 
• :! station and associated antenna, n cable system, or any analogous installation of 
> : equipment for transmitting video streams and accompanying data in analog or 
io digital form. 

:= la most instances, broadcast source 12 will support only uni- directional 

i; communications, and vvdl be used predominately for unidirectional broadcasting or 
u j rnulti -casting to many receivers simultaneously without the receivers* 
loreknowledge. Broadcast source 12 might alternatively support bi-directional 
?! communication. Although only one broadcast source 12 is illustrated for 
5! j. explanation purposes, the broadcasting system can include multiple broadcast 
>7 1 sources, 

; gi Broadcast source 12 includes means for transmitting or otherwise 

is • conveying a primary video stream and ancillary digital data content to receivers 
::;;! 14. In the case of dtgnai satellite transmission, ancillary data can be easily 
:•: i transmitted in digital form along with video and audio streams. Protocols such as 
;i ! .MPEG-2 already provide for incorporating ancillary digital data m packets that are 

downloaded with digital audio-video content using satellite transmission facilities. 

For example, existing D5S systems incorporate "hints'* in MPEG streams to 
■■:[ inotcate aspect ratios tor correct display by receivers. Another example is in the 
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|l analog realm, in which Philips sells a device called the Ghost Echo Cancellation 
;| Reference Signal. By injecting a known digital stream on a .single analog 
tj television V3I Imc, advanced televisions or PCs can more easily detect when a 
ij rnultipath ; ghost; signal is received and take measures to clean up me image. 
1 There are about 800 TV stations in the US that Implements this signal, 
j When using more traditional broadcast, sources, such as analog RF 

! television broadcasts, data can be transmitted in fee vertical blanking interval of 
I! the television signal This method is currently used to provide closed captioning 
| for the hearing impaired. Digital data that has been transmitted in this way can be 
| extracted from a television signal in a number of ways that are known in the 
industry — most currently-manufacrured televisions have the eapsbilitv of 
extracting and displaying closed captioning information. External closed caption 
decoders are -J-r ,*\ai.aKe i -r tnts ourrose \eohtn nai'.y, a numl ?r of external 
devices are available which extract closed caption data and transmit it over a serial 
cable to personal computers or other devices. More recently, it has become 
I possible to decode closed captioning data directly inside a PC using newer video 
I and video tuner boards that allow tor a television signal to be displayed on a PC. 
| There are other mechanisms that are also available for transmitting digital 

i data content to receivers 14. For example, it is possible to use a modern or other 
I conventional network connection for digital data transmissions. Network 
i connections such as these are independent of the traditional video transmission 
media, and have the advantage that they allow bi-directional communications 
between the broadcast source and the receivers. Communication facilities such as 
the internet can also be utilized. Cable television networks might also provide 
1 Similar communications capabilities, in addition, the newest plans for high- 
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: ; definition digital terrestrial TV define channels that are capable of transmitting 
; i 19.3 megabit second of any amount of digital video or data. 

It should be noted that the audio/video stream and the digital data content 
i can be distributed using different media, respectively. While it is convenient to 

■ | ass only a single media such as ovetthe-air transmission, k would also be possible 
■i \\ to distribute the digital data content us:ng a physical medium such as a CD-ROM. 
■: while the video is delivered using traditional broadcasts. Alternatively, both video 
: and data might in some- cases be distributed using a physical, transportable 
j medium such as a CD-ROM. 

■.t in the preferred embodiment of the invention, broadcast source 12 

■ • broadcasts an analog or digital video stream and provides supplemental digital data 
u files to accompany the video stream. These data files have digital content that can 
J3 be rendered by receivers 14, apart from the video stream. The broadcast source 
-.4 ! also provides timing specifications with the supplemental data files indicating 

t<me\ f< -r d,splayirti! trie digital content, relate e to the video stream. 

Trie digital content is preferably audited m a standard, commonly used 
r graphical markup language. In the preferred embodiment of the invention, the 
n supplemental data files are prepared and formatted in a current version of a 

Hypertext -Markup Language (HTML). This allows broadcasters to use standard. 
io widely accepted authoring tools such as used i'br authoring Internet World Wide 

Web content. 

r The supplemental digital data content preferably comprises one or more 

hvperlirsk overlays. Each hyperlink overlay includes one or more hyperlinks. A 
I hyperlink is a displayed region or entity that can be selected or activated by a 
\ s nver " acts r n rxw.d; his a target, indicating some other content { usually stored 



|| in a data files that can. be rendered for the viewer. When a hyperlink $s activated. 
=| its target is retnevsd and rendered. For sample, if the target is a text document 
I; the document js displayed. If the target is a file containing a digitized sound clip. 
!. the .sound clip is rendered. The most common use of hyperlinks is to move or 
| "navigate " from one document or object to another. 

| It. will be apparent to those familiar with Internet technology that the 

I hyperlink overlays described above are similar to documents (often referred to as 
h "Web pages") that can be viewed when browsing the internet's Worm Wide Web. 
jl This is an attractive feature of the invention, because it allows broadcasters to use 
J existing Internet content development tools for desienrae aneilia~v disdtai dai? 
!} content. 

One difference from a conventional Web page is that a hyperlink overlay in 
accordance with the invention has a background that is set to a predetermined kev 
I color. Background areas, set to the key color, are intended to be transparent. A 
: background such as this is easily created, again usmg standard internet content 
j development toots such as Microsoft's Front Fags4 N web designer. 

In this embodiment of the invention, receivers 14 are conventional personal 
computers U?Cs) with equipment having display hardware tor receiving and 
displaying broadcast video streams such as television signals and digital video 
broadcasts. The PC is also associated with a satellite dish 15 for receiving digital 
satellite broadcasts, or a regular antenna for receiving analog TV signals. 
| The display hardware includes bit-mapped graphics capabilities for 

| displaying static bit-mapped images in conjunction with conventional application 
| programs, in addition to video display capabilities. Furthermore, PCs 14 include 
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: '■ video color keving hardware that can be configured to dlsplax video orilv in 
; display ureas that are set to a Key color. 

^ . Alternatively, receivers 14 might comprise oiher types of devices, such as 

.= enhanced television units having capabilities similar to the PC described herein. 
«i As an exar&pie implementation, PC 14 is a personal computer having a 

s| processor (e.g., x86 at Pentium?!) microprocessor from Intel Corporation), 

• i memory, a computer monitor ie.g , VGA. SVGA), and one or mors: input devises 

(e g.. keyboard, mouse, etc.; 
i ■ PC has a digital receiver configured to receive digital broadcast data m a 

u 1 packetned format men as MPEG-enceoed digital video and audio data. Such a 
!: < digital receiver also has access means for reading supplemental data files received 

• ; from satellite broadcasts — -he receiver can receive digital data in many different 

forms. Including software programs, programming informanon. and other ancillary 

data m the form of data files. 

The PC runs an operating system that supports multiple applications. The 

operating system ss preferably a multitasking operating system that aaosvs 
•■■jl simaltaneotus execution of multiple applications. The operating system employs a 
;s i graphical user interface in a windowing environment which presents the 

applications or documents m speciall;>- delineated areas of the display screen called 
xC "windows." One preferred operating system is a Windows® brand operating 
i. system sold by Microsoft Corporation, such as Windows 95 or Windows NT. 
■■■■ W YYmdovt's CE or other derivative versions of vVmdowsvS-. It is noted, however, that 
:: I j other operating systems that provide windowing environments may be employed. 
m ! I such as the Macintosh operating system from Apple Computer, Inc. and the OS/2 



One example implementation of a broadcast-enabled PC is described in a 
co-pending U.S. Patent Application Serial No. 08/503,055. entitled "Broadcast- 
Enabled Personal Computer; 1 filed January 29, .1996. This application is assigned 
to Microsoft Corporation, and is incorporated herein by reference. 

Fig. 2 shows an example implementation of PC 14 in more derail PC 14 b 
enhanced for purposes of displaying broadcast television and accompanying digital 
data content. It includes a motherboard (not shown) having a processor 52 (e.g., 
:<S6 or Pentiumg microprocessor from Intel Corporations and volatile memory 54. 
Volatile memory 54, m combination with non-volatile memory 55, forms program 
storage memory 56 Non-volatile memory 55 comprises a floppy disk, a hard disk, 
a CD-ROM, or some other type of computer- readable storage media. Application 
programs, contatning instructions for performing the steps described herein, are 
contained in the storage media. 

PC 14 includes a digital broadcast receiver $8, such as a satellite dish 
receiver, F..F receiver, microwave receiver, or the like. The digital receiver 53 
receives digital data broadcast over a broadcast network such as a satellite 
network. The receiver 5$ is coupled to a tuner 60 which runes to frequencies or 
channels of the broadcast network. The tuner 60 has one or two primary 
components: a specialized digital broadcast runer and-'or a generalised digital 
broadcast tuner. The specialised digital broadcast tuner is configured to receive 
digital broadcast data in a particularized format, such as .MPEG -encoded digital 
video and audio data. The generalized digital broadcast tuner is configured to 
receive digital data in many different forms, including software programs and 
other ancillary data. 
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H The tuner 60 is connected to the motherboard and data processor :2 via a 

, multi-bit bus 62, such as & 52-bit PCI (Peripheral Component Interconnect) bus, 
i An optional cryptographic desics 64 provides cryptographic services for the cheat, 
such as encryption, decryption, authentication, and digital signing. 

The PC 14 has a video subsystem 66 connected to the PCI bus 62. Video 
and audio data is rran&ierred from tuner 60 over PCI bus 62 to the video subsystem 
66. in some embodiments, raaer hardware instead decodes an MPEG stream itsslf 
without ever passing the data over the PCI bus. In such systems, the 
| uncompressed raw YlTv video is sent along a simple wire connection to the video 
ij subsystem ddboard. In the illustrated embodiment, however, the video subsvstem 
| 66 includes circuitry for decoding MPEG-encoded or other video data formats. 
Video subsystem fid also ;ndndss video display drivers for driving a computer 
monitor 68, 

The video subsystem 66 supports many peripheral devices, in addition to 
the monitor 6S, For instance, the video subsystem 66 might be connected to a 
laser video player "0 tor playing DVD (digital video disks), a game machine 12 
for playing video games, and a VCR (video cassette recorder) 74 for recording 
programs. The video subsystem 66 is adapted for connection to an analog 
broadcast television system 16 to receive conventional TV signals from cable 
television or R.F broadcast television systems. This enables backwards 
compatibility to analog TV systems. 

Video subsystem 66 also incorporates color keying fearures that a<d in 
; integrating video with static bit-mapped graphics. With color keying, a.n 
\ application program or the operating system configures die video subsystem to 
j display a video stream in a rectangular area or 'Viewport" of monitor 6S in 
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cooiuncuon with whatever bit-mapped monitor image has been put in place by 
executing software, in effect the video subsystem overwrites me normal bit- 
mapped display image with the video stream. However, the video subsystem is 
configured to overbite only those areas of the bit -mapped display image that are 
set to a predetermined color or chroma key value. Thus, a bit-mapped image can 
be created having "transparent" areas or regions that are ser to the color key value. 
The video stream will be displayed only in these transparent regions, so thai the 
bit-mapped image will appear to overlay the video stream. 

Tseng Labs, Cirrus Logic, Btooktree, ATI and S3 are examples of 
companies that manufacture video subsystems such as this, 

.Monitor 6S is preferably a VGA or SVGA monitor as is customary for 
personal computers, as opposed to a standard television. In the illustrated 
implementation, PC 14 does not convert the television-related data into an NTSC 
(National Television System Committee) format. Because of this, PC 14 is able to 
produce television data having superior quality when displayed on the VGA 
monitor. Other embodiments might convert images from VGA to NTSC tor 
dispiay on a standard TV screen, although they would have comparatively lower 
graphics reso lutton, 

"Die PC 14 also includes a second bus 130, such as an ISA (Industry 
Standard Architecture) bus, coupled to the motherboard and data processor 51 An 
audio board 13;? is coupled to the ISA bus 130 and serves as an interface with a 
number of audio output devices, such as conventional speakers. An amplifier may 
be coupled between the audio board and speakers if desired. The audio board :s 
also coupled to the video subsystem 66 to receive decoded audio signals. The 
audio board IT.: can be coupled to a stereo system 134, so that audio data can be 
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til output to the stereo system for enhanced sound and recorded. Newer coraouter 

:|1 system? might alternatively support sending digital audio streams outside the PC 

■ ; via a Universal Serial Bus or IEEE139«i connection to an externa; converter. This 

i eliminates the need for an internal sound card and results in much higher fidelity 

5 audio. 

t A CD-ROM (or DVD) drive 136 is coupled to the ISA bus 130, The audio 

: output produced by the CD ROM (or DVD) drive 136 is passed to the audio board 

s 132. 

* ?C 14 includes a modem 13S, such as a 14.4 or 2S.S kbps fax/data modem, 

:o coupled to the ISA bus 130. The modem 13$ is connected to a conventional 

n telephone line and provides access to public networks, including the Internet. The 

; ; modem 1.3S can be used to access and download data and supplemental content 

:js | directly from an independent service provider. It can also be used to download 

-..s ! supplemental data files originating from, a broadcast source. Additionally, the 

:i i modem 138 can be used for r\vo»way communications with the broadcast, source or 

another entity providing broadcast-related services. 

;r An mput'output iVO) adapter 140 is coupled to the ISA bus 130 to interface 

5« with numerous I/O devices, including a digital tape driver 14-2, a floppy disk drive 

| 1-54. and a hard disk driver 146. A remote receiver 148 i 3 also coupled to the 10 

r.o i adapter 1 40 for receiving signals from the remote cordless keyboard 66 and remote 

! control handset 6S in an IR or RF format. Alternatively, the keyboard and handset 
can be directly wired to the computer. The I/O adapter 140 further provides 

z-- !• conventional serial ports, including a COM! port ISO, a COM2 pen 152. and an 

^ i LPT I port 154. An IR transmitter (not shown s can be coupled 10 the COM I port 

;« i 150 to generate infrared signals to control electronic devices, such as stereo 
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» ; shipment. VCR, and the like. The computer 60 can also be hooked directly to 
■ jj these components. 

Ik accordance wsth the invention. PC 14 is programmed or configured to 
-|| receive a digital viceo stream and accompanying HTML files from a broadcast 
: || source, and to display the video stream in conjunction with the hyperlink overlays 
s jj de feed by the HTML files. 

"|| » shows how this is accomplished- An overlay window wl) is opened 

■s|| tor displaying a hyperlink overlay. Video subsystem 66 is configured to display 
•j| the vid£0 stream ?n a v^wport 201 that coincides in size and position with the 
.o| overlay ^in-iow 200. The vmeo subsystem is also ccrdlgured to display the video 
• •I scream only over those areas of the hyperlink overlay that are dot set to the 
j: predetermined color key. Thus, non-transparent areas of the hyperlink- overlay 
appear "through'' the video stream, and appear to overlay the video stream, 
•■i Perhaps the best example of this behavior is the TV weatherman who appears to 
■ : "float" in front of 3 wsather map. A same window 202 is also utilized to define 
:i\ the edges of overlay wmdow 200 and viewport 20 L These three components are 
-i configured to appear as a single window- to a user. Operating software is 
is ] configured so tha* fee user can resbe and move the components as a snrgje emuy. 
:«| Overlay *mdow 200 is preferably implemented using an ActiveX™ 

,:o | control that is desiped specifically for displaying an HTML-formatted document. 
| Such an ActiveX" M HTML control is configured to also allow browsing or 
navigating among documents by activating hyperlinks. ActiveX^ 5 is a standard 
for interchangeable components that has been defined by Microsori Corporation. 
^ ij ActiveX™ controls ire display entities that allow a user or viewer to interact with 
„■ | programs through the display, in this case, the control displays an HTML- 
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: : i forma rted document, allows rhe user to activate hyperlinks, sad retrieves and 
:l; displays me targets oi such hyperlinks. A conventional HTML browser can 

• ahemaus ety be used lot displaying the hyperlink overlays. 
^| Rather than associating displayed icons with hyperlinks, it is also possible 

?! 10 create a totally transparent itnagema.sk rasing HTML terminology; to be 

overlaid on top of the video. This results in '"hot spots" on the screen that can be 
-I clicked oo for an acnon to occur. An example of this might be to create a hotspot 
j | on top of an actor's body; disking on the hotspot would reveal what the character 
•j I was th.in.king. Another example: a user could chck on Tim AlkrAs Binford saw 

during an episode of 'Tool Time" to find out more information about ir such as 
\ i where to purchase it. 

Hyperltnk overlays can be provided in groups corresponding to a particular 
;i , video stxearm allowing navigation among various individual overlays of the group, 
:A For example, a rdated group ot overlays might be provided to a PC prior to the 
:s| beginning of a particular episode of a weekly television show. The first of the 
is i overlays would be displayed at the beginning of die episode, and the viewer could 

then follow hyperlinks as desired to view -he other overlays of the group, 
is Aitemanvclv. timing and other information can be provided to the PC along 

>v with communication packets in which the overlays are transmitted, lite timing 
io information indicates times, relative to the video stream, at \0uch particular 
z\ k overlays should be displayed. Hyperlinks can be activated to override the 

specified timing. An alternative method to synchronize the video that doesn't rdy 
upon specific timing relative to the video stream Is by using closed-caption text. 
;i Bv examining a closed -caption stream, specific events can be triggered to occur 
„; : based noon v.- her. the actors on trie screen emmciate certain words that serve as 
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1 reference marks. rhick technology is described m eo- pending ,.'S patent 
[' app:icaaun Sena; \'o 0$. 7.9. 2 TO, rati-Jed 'System arid Method For 
■: Synchronizing Content V.'ith A Video Program Csmg Closed CiptiorungC riled 
: : January o, 199" This patera application, which is assigned to Microsoft 
Corporation, is hereby incorporated by reference. 

PC 1-i uses a stack-b.ia5.-d algorithm to handle multiple overlays, so that one 
o^-erlav can be temporariK ovenvrirter: by another. For example, an ememencv 
news transmission msgrht have overlays dun would overwrite a regularly scheduled 
show Wlien the emergency new?; transmission is complete, trie previous overtavs 

|j Fig. ■■'; shows preferred methodological steps performed by broadcast source 

: 12 in accordance w-.xh the invention. A step 220 comprise? transmstnng a video 
1 sneam. The video stream can be m one of a * artery ox formats, such as a 
jl traditional analog RF television format or a digital formal originating from a 
■= satellite or a cable headend. 

A step 222 ,(?rnpriifv fbrrnamng supplemental data files m a graphical 
markup language, preferably HTML, Each supplemental HTML data die has 
instructions tor rendering a h\-perkni: page or overlay on the video stream at an 
indicated time. Each overlay can have transparent areas or regions, which are set 
to a key color or chroma color. CK-enays can also contain hyperlinks to other 
I overlaps and to other non-overlay documents and objects. 

I A step 22-* comprises associating the supplemental data files w s tb die video 

| stream. This step ss performing bv speeiryrng control data such as timing 
I parameters along vsith the video stream, indicating t-rr.es lot displaying the 
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;>or* nror \i ise l:rne thai thev will be needed, taking data transmission scesd into 

yter „Z6 comprise? transmitting the supplemental data files alone v^th the 
xidfo stream As already discussed, this can be accomplished using rhe built-in 
- , capabilities of the digital transmission media or using the vertical blanking interval 
<, || of a tractitiQaa. analog television signal. 

-;| Fig. 5 shows steps performed by PC 14 or another receiver. A step 230 

V comprises receiving the video stream and accompanying supplemental from the 
vl broadcast source. Step 232 comprises displaying the video stream on a display 
device in a viewport using color keying technology, so that the video stream is 
:i j; actually displayed only on areas ot the dtsplay thai are set to a predetermined key 
; ; •. color or chroma color. 

:: j A step 234 comprises displaying the hyperlink overlays, defined by the 

: Jj supplemental data hies, on the display device m conjunction wnh tbe video stream. 
~ ■! If the overiavs ,~ave been previously cached, this step includes revrieving ihein 
-J: from cache storage. Control daw., received along with the supplemental data tiles. 
:> jj is used in this step to associate hyperlink pages or overlays whs the video stream, 
■s,. As a result, overlays are displayed at the times indicated by the timing 
;<,!! specifications contained in the control data, rather than at the nme they are 
>o II received. 

; .,|| in the default case, an ActiveX™ control is used to display the hyperlink 

overlays and to ailow navigation using hyperlinks in the overlays. The ActiveX™ 
- : control is ceniiaureu to occupy the same area as the video stream viewport. Thus. 
\ ; || the video stream is rendered in the transparent areas of the hyperlink overlays In 
. < some case*. PC ;c rrught actually create or render the ActiveX™ control arieae or 



1? 

=1 time, although *he control n not mace vjsibk ur.nl the appropriate time. Tins 

: ; avoids any delay sr. aresentmg overlays. 

■., Alternatively, comcl data might define multimedia objecrs that are not 

-]| transparent hyperlink overlays. When this is the case, a separate application 

; || program is launched, such as an HTML -compatible browser, to render the object 

■i ; defined by a supplemental data file. Such an application program is launched in its 

■ || own. independent window, and in the case of visually-oriented objects, can be 

* |j viewed alongside the video stream. 

*j! As aotcd, control data, tnrnsrnirreo" along with the supplements! data files* 

jj indicates how the snDplemsnta; data riles should be handled. Primarily, fee control 

;; j data indicates times at which the data files should be rendered or made visible, la 

;; j addition, the control data indicates what should happen when a viewer activates a 

:j hyperlink from within, an overlay. Generally, activating a hyperlink causes its 

• , target to be displayed or otitenvi.se rendered. However, there are rsvo options when 

.? : a hyperlink's target is another transparent overlay; the target can either replace the 
currently displayed overic> or the target can be displayed by &n independent 
browser in a new window that is opened just tor this purpose. If the h\psriktk 5 s 

is target is not a transparent overlay, there is only one choice: the target is displayed 

■ -■> ! in a new window by launching an appropriate application program. Step c'lh* thus 

x\\ includes a step of launching application programs as required to render non- 

:: ] \ overlay content targeted by hyperlinks. 

.v. ij In the PC embodiment described above, these steps are penbrmed under the 

;j |i control of data processor 52, during execution of programming instructions that are 

:i i' stored in program memory 56. 
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■ I VChlk the invention has been described prima rily for use with broadcast 

: ; video streams. now thar video streams might also be available, through other means 
: } such as local storage (hard disk. DVD, CD, VCR, etc. K The invention is also 
*j useful in conjunction with video sources such as these. For instance, a movie 

might be provided on a DVD along with a group of transparent HTML overlay 
r I files for use along with the movie, Alternatively, the movie might be provided on 
> ; a DVD, wrth the overlay files being downloaded from art internet source. 
>| Also note that even with broadcast sources, the transmission of 

s ; supplemental dais files does not have to rake place conciirrendy w-th transmission 

of video streams. Rather, in some systems it might be desirable to broadcast 
: ; | overlay files during she night to user equipment, for use the following day. 

Control data indicates which programming the overlay tiles are to be associated 

- j i with, and software at the user's PC retrieves the appropriate files at the appropriate 

• times depending on the viewing selections of the user. 
;/!; The invention provider an easy way for broadcasters and other v-deo 

a\ providers to supply supplemental textual and graphical materials, and to have such 

H materials coordinated whh video content. In contrast to prior systems, the 
:s| invention takes advantage of widely-used hypertext authoring packages, as thus 
: « I encouraging acceptance and standardization 

;5 ; hi compliance with the statute, the invention has been described m language 

;■!! more or less specific as to structural and methodical features. It is to be 
understood, however, that the invention is not limited to the specific features 

- ;= described, since the means herein disclosed comprise preferred forms of putting 
-J: the invention into effect. The invention is, therefore, claimed in any of its forms or 
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CLAIMS 

L A method comprising the following steps. 

fcrmatrm$ supplemental data tiles m a graphical markup language, each 
supplemental <hsa file having tr.structions for rendering a hyper lirJ: overlay on the 
video stream: 

nansnutung me supplemental c;ua files along with the video stream. 

2, A rrera.«! a* tec.teo .r ehim: 1 * he; em tre lurmanm,; ;tep comprises 
tc-rnunmc the jop^Vrrseru data fdes *n. i,7\1L 

3. A m.smod a* seciteu in escum 1 anc farther co:srpr.<mg a ^ter < ,v 
trautunittsrig urnm? sre , e:tl:,anom v. the supplemental oar^ :i ei .mv„ar;ng 

for displaying the hypcrimk overlays. 

4. A computer-readable storage medium having computer-executable 
instructions for perionnirtg steps as recited in claim i . 

5, A method as recited in claim } arsd further comprising the following 
additional steps: 

rccers frig, the vtdec jrream and accompanying supplemental data files: 
displaying fhe hvperhrsk overlays in cord unction with the video stream. 
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o. A mctr t o-i as recited in ohsm 5, wheiem ihe dispjaym^ step corr.pnsei 
launching an liTMl.-ccmpa'ibk browser to d^phy the hyperlink overlays. 

7, A method as recited in claim 5. wherem: 

the fonnamsg step comprises setting transparent areas of each hyperlink 
overlay to a key color: 

the 'hsplv mot comprises d:s,pL:\i\s :*e vide- j»cie--n oris t\e area? 
of the hype: 1 nk e\ e-u^ :!x; ^e v t :o . l< \ ccvk\ 

8. A n-L-oo rv\i1 *r, a «ren. 

the fornawas step cotnpi^if* wit^ nraaspaicn: .atfjtf ^*\acr per irk 
overlay to a *eev color: 

he * -rep o.*n>nsrs dur^Tr: HTML C0;roat -^e n~o\\3« i*» 
disp ne "Te-f^x^-, .:ic * 

t.ie cs ^vr , .-ther vcrnpvses vL*~ av \ico or ear? ^ 

the << 4? Kre- ir\ r\e- "v *>a: ^j** r* e> jl *> «> \or 

3 ' rre< *\V a-< ,ceKC^ .j; c^un o h mere ~i 

Mm ,.-C"U)ii~^ <=*Tjr-_* -in^rar^r atcr& ot r^-imc 

overlay io a key colon 

co r-.ee k , aeol^osv e hot ^pr\$' ~e ~j aivpux Wi;ut*a. e 

a key color. 
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1U, A method as ;ecned m claim i. v. herein: 

the formatting step comprises netting, transparent areas of the hyperlink 
overlay to a key colon 

the displaying step comprises launching an HTML-compatible browser to 
display the hyperlink overlays; 

the displaying step further comprises osmg color keying vjdeo hardware 
that displays video omy in display areas that are set to a key color. 

1.L A method as recited m claim 5, wherein the formatting step 
comprises including hyperlinks m the hyperlink overlays, die method further 
comprisin.fi an additional step of displaying content targeted by such hyperlinks m 
response to selecting such hyperlinks-. 

12, A method as rscued jr. chin: 5. wherein the formatting step 
comprises iududmg hyperlinks in the hyperlink overlays, the method fhrrher 
comprising an additional step of replacing any currently displayed hyperlink 
overlay wnh content targeted by such hyperlinks in response to veieetmg such 
hyperlinks. 

O. A. method as recited in claim 5, wherein the formatting step 
comprises including hyperlinks in the hyperlink overlays, the method furdier 
comprising an additional step of opening new viewing windows lor displaying 
1 content targeted by such hyperlinks. 



comprising an additional step of launching application programs as required so 
render content targeted by such hyperlinks. 

15. A computer-readable storage rnedrum having cotnputer-execatabie 

instructions for performing steps as recited in claim 1 

16. A. method comprising the following steps: 
transmitting a video stream; 

formatting HTML files having instructions for rendering hyperlink pages, 
the hyperlink pages having transparent areas that are set to a key color: 
associating the HTML files with the v>deo stream; 
displaying the hyperlink pages on a display; 

displaying the video stream on the display in areas of displayed hyperlink 
pages that are set (o a key color. 

H« A method as recited in claim 16, and further comprising the 
followmg additional step;;: 

transmitting timing spectilcations with the HTML files indicating times for 
displaying rhe hyperlink pages; 

the step of displaying the hyperlink pages being performed at the dines 
indicated by the timing specifications. 



1&< A :iet>oc a-< r*cued <r. c -jrr It w:erem ne step o: o.-sr as ng the 
h^peslmk papt-c c^-tier: ..u:r.emr.g H 1 VL cor.oro... k l<:>_v.ser 

19, A method as recited in claim 16, wherein the step of displaying the 
video stream comprises using color keying Video hardware that -displays video 
only in display areas that are set to a key color. 

20v A method as recited in claim 16, former comprising an additional 
!; step of displaying content' targeted by hyperlinks in response to selecting such 
hyperlinks. 

21 - A method as recited in claim Id. further comprising an additional 
step of replacing any currently displayed hyperlink pages with content targeted by 
hyperlinks m response to selecting such hyperlinks. 

22v A method as reened b claim 16. the method further comprising an 
additional step of opeaing new viewing windows tor displaying content targeted 
by hyperlinks m displayed hyperlink pages. 

23. A method as recited in claim 16 : further comprising an add! -tonal 
step of launching application programs as required to render content targeted by 
hyperlinks in displayed hyperlink, pages. 
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associating one or more hyperlink pages with the video stream, the 
hyperlink pages baling transparent areas that are set to a ksv color: 
displaying the hyperlink pages on a display; 

displaying ibe video stream on the display in areas of displayed hyperlink 
pages that are set to a key color, 

25, A metnod as recited in claim 24, wherein the step of displaying the 
hyperlink pages comprises launching an HTML-compadbisr browser. 

A irei^c; j detect tn . a.rt IX v. •vera the *tep ^ iisplaving oc 
video >„rearn cc-p™^, evre color keying vidsc naruwate ram to configured ro 
display \ jd~r o„v ,r. it-p^ ate as char ait; -ei i n & ke\ color. 

2". \rrevo - ^ d <n ~ia ~n - * ^ne^r 

"e step e: ix-r\£- »rs ne -r^otr .ik pare*- ^-mrrtsce ,aL-vr nz m Ml - 
compatible browser; 

the step of displaying the video stream comprises using color keying video 
hardware that :s configured to display video only in display areas that are set to a 
key color. 

2%. A method cs iccued :n c[an\ c*4 further eofrxn^rx an additional 
srep of dli-pLy ins tonren; targeted k\ ^-psriinfcs vr. resoorwe to selecting such 
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29, A method as recited in dairy- 2-4. further comprising an additional 
step of replacing any currently displayed hyper iink pages with content targeted by 
hyperlinks in response io selecting such hyperlinks. 

30, A method as recited in claim 24. the method further comprising an 
additional step of opening nesv viewing windows for displaying content targeted 
by hyperlinks in displayed hyperlink pages. 

31- A method as recited in claim 24. further comprising an additional 
step of launching application programs as required to render content targeted by 
hyperlinks in displayed hyperiiak page?. 

32. A. cor",pv i :er-"eai>*oie i,:,?rigc raeoaum. Laying computer-executable 
mst: actions lor reforming ^epa ^npriv^ 

raeeo 'i- sLoanx 

ass ux."; - s r<e c ircxe Nvuirx pvses ^ xb rhe video stream, die 
t, l^Tcrl nk ragus *>avnif j^seaient areas nat aie set to o key colon 
display ng trie Irq ed ^ pages on a oispsav: 

dtsc.^ng wde ottsam on tnc display s n areas of displayed hyperlink 
races that ~.te -ei V a \ ev <~ Jot 

33. -\ <. .mpuc. iCid-l'le storage medrcri as recited m claim 32, 
o ncem *he i*v.p ot J'spL\ .n: toe nvpirnnk ytj**.; :nrrprises launching an HTML- 
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34. \ oorvriues -.sartudv storacie r.ccLarr a- r^r.rc >r o!am 
vhenn tl~e *\ 4 p c; c.s.x-av*. the y.je^rr. ocnpr-Vi ucir^ co,~r Lc^'x^ 
x -de; i^d^src r.a; o ^n;1gur?d ; s d?ar/,u % .oeo cr.v n cL*ivis ..reas t.us. .srt 

is to a kev color. 

35. \ oomousr roa;i"lc stoiagi snedrcu as ^so.teo .\ v arrs 
n wherem: 

ine step cf c sclv mg ;r.e rvro^iric r\~es comrrscs laur^nir.g an H'MI - 

t^.e ^ . i i^p av.ni v .<5eo STcm ; :i:i-^t^os usms .^.or s.oy.u^ v ~deo 
hari^a^e tiaf cor.fi.Tv~v - to d^rslav vtdf oru> :r. di*ria\ areas t.,at v-e * a x ;~. a 
x:^ l.j!. n. 

36. A eoir.pvHer-reiidafois stooge medium as recusd in chun 32 having 
further computer- executable instructions for perferrndig the additional .step of 
displaying coatem targeted bv hyperlinks in response to selecting such hyperlinks. 

37. A computer-readable storage medium as seated in claim 52. having 
turther computer-executable instructions for performing the additional step of 
replacing any currency d.spiayec hyperlink pages wuh content tested by 
hyperlinks in response io selecting such hyperlinks. 
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: j 38. A compmer-reacjabie storage medium as recited in claim 32. having 

: I further computer-executable in struct; ons for performing the additional step of 

j opening new vievnng windows for displaying content iarssted by hypenlnks in 

.5 1| displayed hyperlink pages. 

J 39- A computer-readable storage medium as recited in claim 52, having 

■■! further computer-executable instructions for performing the additional step of 

si launching application programs as required to tender content targeted by 
hyperlinks m displaces hyperlink pages. 

: ; lj 40. A video broadcast system comprising: 

. : i a broadcast source dial broadcasts a video stream and provides 

s: | accompanying supplemental daia files, each supplemental data ale having 
.4 instructions for rendering 2 hyperlink overlay on the video stream: 

a receiver configured to receive the video stream and accompanying 
>a\[ supplemental data files and to display the hyperlink overlays sn conjunction with 
i? I the video stream. 

;? 4L A video broadcast system as recited in claim 40. wherein the 

2C . supplemental data nles are icrmaned in HTML,, 

r '| 42. A video broadcast system as recited in claim 40. wherein the 

:: < [i broadcast source provides timing specifications with the. supplemental data files 
si indicating times for displaying the hyperlink overlays relative to the video stream. 
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43. A video broadcast sysretr; as rec ued in claim 40, further comprises 
an BT>lL-coni.psnb.e browser that the receiver launches, to display the hyperlink 
overlays. 



s:j 44. A video broadcast system as recited in ciatm 40. wherein the 

* 1 1 receiver comprises color keying hardware tbar displays video only in display aress 
- 1 1 that are set to a key color, the hyperlink overlays having iranspar&at areas that are 

s j set to a key color. 

•i> 45. A receiver for receiving and displaying video streams, comprising: 

: ; display hardware for displaying video streams and bit-mapped images to a 

user, 

:i the display hardware Including color keying hardware that disalavs video in 

h ij display areas that are set to a key color; 
h access meacs tar reading supplemental data files that have insrrucnons for 

rendering bit-mapped hyperlink overlays tn conjunction with the video stream at 

:% a data processor that reads the supplemental data files and m response 

displays the hyperlink overlays at the indicated times, wherein the hyperlink 
;c>, overlays have transparent areas that are set to a key color, the hyperlink overlays 
~ : j thus appearing to overlay the video streams. 
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46. A receiver as recited ;r- claim 45, further ccmpnsine an HTML- 
compatible browser that the data processor executes to display the h>perlink 

: 47. A receiver as recited in claim 45, svherein iht data processor is 

} programmed to display content targeted by hyperlinks .in the hyperlink overlays in 

j response to selecting such hyperlinks. 

: 48. A receiver as recced in claim 45. wherem the data processor is 

! progmntrned to replace any currently disptayec hyperlink overlav with content 

i; targeted by a hyperlink in said currently displayed hyperlink overlay m response to 

j selecting such hyperli.ck. 

49. A receiver as recited in claim 45. wherein the data processor is 
: programmed to open a nesv viewing window for displaying content targeted by a 
II particular hyperhnk m response to seiecrmg the particular hyperlink. 

j: SO, A receiver as recited in claim 45. wherein the data processor is 

i programmed to launch application programs as required to render content tatgeted 
I by hyperlinks in hyperlink overlays. 



Office 
il 

Application No: GB^MM4.7 Examiner: R F King 

Claims s*arrit«l; a j » a te of sesircii: 10 November \m 

Fatextts Act 197? 

Search Report under Section 17 

Databases searduxk 

UK Patent Office collections, wetodmg OB, H\ WO & US natem specifications, ;ri: 

UK CI (bd.P): H4FEFBB], H4T£TBAX. TBBN, TDAA] 

intCHK.6}. Ci06F 17/30 
Oth er: ONLINE: INTERNET, WPI 



DoajBsents cmtsidered to ht relevant: 



Category 


identity of document and relevant passage 


Reisvaat 


A 


OB 2 309 1 34 A [Concept Dev.] See abstract 


1, 36, 24, 
■SO, 4;i 




